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EXECUTIVE  SUNNARY 


There  is  probably  no  problem  more  basic  to  Program  Managers  than 
cost  estimating.  Current  decisions  l'egarding  future  courses  of  action 
require  good  estimates.  To  predict  new  costs,  different  techniques  and 
methods  have  been  used.  However,  they  have  all  shown  the  same  common 
characteristic  --  none  has  any  guarantee  of  success. 

In  order  to  improve  the  overall  management  of  1*0 fens e System 
Acquisition,  the  Department  of  IAafense  established  the  Defense  System 
Acquisition  Review  Council  (DSARC)  within  the  Office  of  the  Secretary 
of  Defense.  This  Council  was  to  evaluate  the  status  and  readiness  of 
each  major  weapon  system. 

Past  studies  proved  that  weapon  systems  cost  growth  was  a result 
of  low  initial  estimates.  In  order  to  improve  upon  this,  the  Cost 
Analysis  Group  fCAIG)  was  established  by  the  Secretary  of  Defense. 

Ihey  were  to  review  and  provide  the  DSARC  with  both  program  costs  and 
ranges  of  uncertainty  within  which  costs  would  likclv  be. 

In  addition,  the  Department  of  Defense  desired  that  the  "par;imetric 
approach"  for  weapon  system  cost  estimating  be  used.  There  is  no 
doubt  that  the  parametric  approach  or  as  they  are  so  often  called,  cost 
estimating  relationships  (CP.les),  can  be  of  great  use  in  cost  estimating. 
Ihcir  capability  to  produce  accurate  estimates  is  limited  and  should 
be  recognized.  Viewed  realistically,  they  are  useful  means  of  combining 
limited  information  to  provide  "ballpark"  estimates,  since  in  many 
cases  they  cannot  capture  all  the  variances  in  cost,  Ihcrcfore,  estimates 
derived  by  the  parametric  technique  should  lie  used  with  caution, 


particularly  when  extrapolating  to  distant  future  weapon  systems. 


If  viewed  as  a point  of  departure,  and  modified  by  experience, 
judgement,  and  external  or  supplementary  information,  they  can  he  used 
extensively  with  much  success. 
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INTRODUCTION 


During  past  years,  the  cost  growth  in  System  Acquisition  has  in- 
creased to  such  an  extent  that  it  has  become  the  subject  of  national 
debate.  With  massive  cost  overruns  in  major  weapons  acquisition,  as 
evidenced  by  mismanagement  and  waste,  the  Department  of  Defense  decided 
that  attention  must  be  given  to  improve  the  management  of  Defense  System 
Acquisi tion. 

In  May  1969  the  Assistant  Secretary  of  Defense,  David  Packard, 
established  the  Defense  System  Acquisition  Review  Council  (DSARC)  within 
the  Office,  Secretary  of  Defense.  The  sole  purpose  of  this  Council  was 
to  advise  him  of  the  status  and  readiness  of  each  major  system.  The 
membership  of  the  Council  consisted  of  the  Director,  Defense  Research 
and  hngineering,  and  the  Assistant  Secretaries  of  Defense,  Comptroller, 
Installation  and  Logistics,  and  System  Analysis.  The  Council  was  to 
evaluate  the  status  of  each  candidate  system  at  the  following  three 
milestone  points: 

a.  Program  Initiation  decision,  i.e.,  when  the  sponsoring  Ser- 
vice desires  to  initiate  Contract  Definition. 

b.  Pull-Scale  lAvelopment  decision,  i.e.,  when  it  is  desired  to  go 
from  Contract  Definition  to  full-scale  development  and, 

c.  Production  decision,  i.e.,  when  it  is  desired  to  transition 
from  development  to  production  from  service  deployment. 
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The  Program  Manager  is  caught  in  a vise.  The  programs  that  he  is 
developing  tend  to  be  more  innovative  and  more  complex  than  past  pro- 
grams and,  therefore,  more  difficult  to  cost  accurately.  At  the  same 
time  the  frequently  casual  costing  concepts  of  the  past  are  increasingly 
unacceptable  to  the  administration,  the  Congi'ess  and  ultimately  to  the 
people.  Fortunately,  presently  available  methods  of  cost  analysis  make 
it  possible  for  the  informed  manager  to  develop  and  substantiate  accurate 
projections  of  program  costs. 

Early  in  1970,  the  Assistant  Secretary  of  Defense  for  System  Analysis 
wrote  a memorandum  clarifying  the  OSD  position  on  the  use  of  statistical 
techniques  for  weapon  system  cost  estimating.  This  paper  was  circulated 
throughout  the  Army  as  an  inclosure  to  a memorandum  on  the  use  of  sta- 
tistical techniques  in  cost  estimating.  It  states  that  estimates  for 
new  weapon  system  acquisition  cost  were  to  be  either  derived  from  de- 
tailed, grass  root  calculations  (the  industrial  engineering  approach)  or 
based  on  relationships  between  more  aggregate  components  of  system  cost 
and  the  physical  and/or  performance  characteristics  of  the  system.  These 
so  called  relationships  were  to  be  derived  from  cost  histories  on  prior 
programs.  11 '.is  method  is  often  called  the  "parametric  approach."  This 
type  of  estimating  is  used  during  early  phases  of  system  acquisition 
since  many  programs  have  limited  and  uncertain  information  on  which  to 
base  estimates.  The  parametric  approach  is  particularly  suited  to  making 
estimates  based  on  limited  physical  and  performance  information.  The 
objective  of  parametric  estimates,  or  as  they  are  so  often  called,  cost 
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estimating  relationships  (CERs),  is  to  provide  for  overall  better  pro- 
gram management. 


The  cost  growth  studies  conducted  by  System  Analysts  showed  that 
most  of  the  so  called  cost  growth  was  the  direct  result  of  low  initial 
cost  estimates  and  the  indirect  result  of  changes  in  schedule,  quantity 
and  rate  of  production. 

As  the  cost  increases,  the  production  quantity  is  accordingly 
reduced  to  stay  within  the  total  program  dollar  thresholds . However, 
due  to  the  learning  curve  phenomena,  the  reduction  in  quantity  further 
inci'eases  the  unit  price. 

A plot  of  the  average  cost  growth  f CHART  I),  as  a function  of  time 
from  the  34  systems  now  in  procurement  and  reported  in  Selected  Acquisi- 
tion Report  (SARs)  shows  that  two  years  after  the  planning  estimate 
was  calculated,  the  development  estimate  surpassed  it  by  40$.  In  June 
of  1971  and  December  of  1972  cost  growth  was  well  over  100$  on  major 
systems.  This  is  a factor  of  2.4  to  the  planning  estimate  in  terms  of 
average  growth  and  some  growth  was  as  high  as  800  or  900'.  l’rocurement 
cost  growth  alone  increased  to  450-500$  and  on  some  piv.jm.s  has  readied 
the  800$  mark.  There  is  great  concern  over  the  initial  low  program 
estimates  for  as  the  Services  continue  to  underprice  the  five  Year 
Defense  Flan  (FYDF) , imbalanced  programs  result.  Therefore,  it  is 
impossible  to  make  resource  tradeoffs  and  further  purchase  the  right  types 
of  equipment  if  programs  are  continually  under  estimated.  The  end  result 
is  that  in  later  segments  of  the  program,  many  important  components  are 
eventual ly  d i searded. 
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In  addition,  if  Services  continue  to  increase  estimates  in  their 
SARS  to  Congress,  credibility  of  any  correct  estimate  will  be  further 


lost.  For  example:  if  one  of  the  Services  arrives  at  a one  billion 
dollar  cost  estimate,  Congress  will  automatically  think  it  should  be  two 
billion  because  of  past  miscalculations.  Hence,  the  Services  are  left 
wondering  whether  or  not  to  over  estimate  since  there  is  no  credibility 
with  their  estimates. 


A STPP  IN'  TUP  RiCHT  HI RPCTION 

To  counteract  the  previously  mentioned  credibility  gap  much  more 
realistic  estimates  are  needed.  On  December  7,  197],  the  Deputy,  Sec- 
retary of  Defense,  directed  Service  Secretaries  to  improve  their  capa- 
bility to  perform  independent  parametric  cost  analysis  and  to  have  such 
an  analysis  accomplished  on  cadi  major  weapon  system  at  eadt  key  decision 
point.  "The  memorandum,  Use  of  Parametric  hsti mates,  advised  that  by 
January  1972  an  independent  cost  analysis  was  to  be  incorporated  in  cadi 
DSARC  presentation.  There  was  a definite  need  for  timely  realistic  cost 
estimates  for  TOD  management.  From  these  realistic  cost  estimates  major 
decisions  could  be  better  employed." (9: 2)  Chart  2 depicts  the  sequence 
of  independent  cost  review  for  DSARCS. 

Past  cost  estimates  were  usually  based  on  the  industrial  engineering 
approach  whereby  extensive  system  descriptions  and  design  were  required 
in  addition  to  a detailed  simulation  of  production  operations.  This 
approach  previously  ment ioned  as  the  "grass  roots"  approach  used  stan- 
dards built  from  time  studies,  motion  studies,  vendor  quotes,  etc. 

Because  of  incorrect  cost  estimates,  program  consequence  included: 

a.  A choice  of  higher  performance  designs  than  were  cost  effective. 

1).  Many  attractive  alternatives  were  foregone. 

c.  Cost  growth  was  destroying  the  budget  by  stretch-outs  and 
fewer  more  expensive  units. 

d.  POD  credibility  was  reduced,  and 

c.  Overall  management  became  difficult. 

Another  ippromh  to  cost  estimatic  > i as  needed  if  the  acquisition 
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of  major  weapon  systems  was  to  survive.  As  a check  to  the  "grass  roots" 
approach,  DOD  decided  to  place  increased  emphasis  on  parametric  cost 


estimates.  These  estimates  were  to  incorporate  costs  of  previous  system 
setbacks  and  present  aggregate  level  estimates.  They  included  actual 
relationships  between  cost  and  physical  performance  characteristics  of 
past  systems. 

In  a memorandum  dated  25  January'  1972,  Secretary'  of  Defense  Melvin 
Laird  established  the  OSD  Cost  Analysis  Improvement  Group  (CAIG)  to  act 
as  an  advisory  board  to  the  DSARC  on  all  matters  related  to  cost.  Mem- 
bers of  this  group  included  representatives  from  DDR^E,  ASD(C) , ASl)(IfL) 
and  ASD(SA).  The  CA I G was  directed  to  review  sendee's  independent 
estimates  and  report  any  resemblance  of  the  cost  estimate.  Service 
Secretaries  were  encouraged  to  provide  staff  components  capable  of  pro- 
viding independent  parametric  cost  estimates  (ICE). 

\ With  this  CAIG  cost  data  report  tiro  DSARC  could  compare  Contractor, 
Service  and  OSD  program  costs  as  illustrated  on  Chart  3.  Ihc  DSARC  not 
only  was  provided  with  a point  estimate  of  most  likely  cost,  but  a range 
of  uncertainty  within  which  the  cost  would  likely  be.  This  range  of 
uncertainty  shown  on  (hart  4 is  necessary  because  all  cost  estimates  are 
only  predictions,  and,  therefore,  contain  some  degree  of  uncertainty. 

Cliart  3 illustrates  the  ratio  of  (L\IG  co  t estimates  to  those  of 
Service  independent  studies  and  accepted  by  CAIG.  It  would  appear  by 
this  chart  that  the  CAIG  cost  estimates  are  significantly  higher  than 


those  of  the  Services.  Therefore,  analysis  of  this  has  proven  that 
during  the  past  few  years  the  CMC.  estimate  has  been  closer  to  the 


actual  cost  of  military  programs,  whereas  the  Service  estimates  have 
been  far  below.  Chart  6 clearly  substantiates  this  and  further  indicates 
that  costs  which  are  reported  to  Congress  in  the  SARs  are  beginning  to 
approach  the  parametric  cost  previously  estimated  by  the  CAIG. 
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DATA  COLLECTION 

As  previously  stated,  a cost  estimate  is  a judgement  or  opinion 
regarding  the  cost  of  an  object,  commodity,  or  service.  "The  main 
problem  facing  the  military  cost  analyst  is  to  develop  and  apply 
concepts  and  techniques  for  assessing  the  economic  cost  of  proposed 
alternative  future  actions  under  conditions  of  uncertainty."  (1:64) 

Tiie  challenge  is  to  project  from  the  known  to  unknown  relationships, 
using  explanatory  variables  such  as  weight,  speed,  power,  frequency  and 
thrust  to  predict  cost  at  a higher  level  of  aggregation. 

Parametric  cost  estimates  have  proven  to  take  less  time  and  have 
been  more  accurate  than  the  industrial  engineering  process.  "One  reason 
for  this  is  simply  that  the  whole  generally  turns  out  to  be  greater 
than  the  sum  of  its  parts."  (3:33)  This  method  involves  the  use  of 
quantitative  techniques  which  are  reproducable.  Thus , a given  tech- 
nique, although  perhaps  not  optimum,  will  always  give  an  estimate  that 
will  not  vary  with  the  particular  person  using  the  estimate.  This 
process  involves  locating  physical  or  program  characteristics  that  have 
a definite  relationship  to  cost.  These  relationships  arc  simple  state- 
ments indicating  that  the  cost  of  a commodity  is  proportional  to  the 
weight,  volume,  area,  or  some  other  physical  characteristic  of  that 
commodity.  The  cost  estimating  relationship  may  also  be  indirectly 
determined  such  as  the  number  of  various  size  uniforms  required  would 
be  a function  of  the  number  of  personnel.  Per  proper  development  and 
evaluation  of  cost  estimating  relationships,  some  familiarity  with 
regression  theory  is  necessary . However,  it  , ist  be  e,  ; 'as  i zed  that 


that  regression  analysis  does  not  offer  a quick  and  easy  solution  to 
all  the  problems  of  estimating  cost.  There  is  no  substitute  for  under- 
standing! The  important  element  is  to  identify  variables  so  that  when 
weighed  and  combined  in  appropriate  mathmatical  expressions,  an  accui'ate 
cost  estimate  will  result. 

In  order  to  provide  an  accurate  estimate,  a requirement  for  a 
systematic  collection  of  data  and  information  is  a prerequisite.  Mili- 
tary cost  analysts  are  no  better  than  the  information  or  data  that  are 
part  of  their  analytical  effort,  hue  to  inconsistency  in  data,  the 
analyst  has  had  to  use  contractors  records  in  the  past.  To  alleviate 
this  problem,  the  Cost  Information  Report  (CIR) , was  established  in 
1966. 

Even  with  the  use  of  the  CIR  much  inconcistency  was  found  in  the 
data  which  it  provided.  The  result  was  the  Contractor  Cost  Data  Report 
(CCIJR) , a new  requirement  of  the  CMC  which  further  standardized 
both  data  and  format  reporting.  "This  report  provided  for  collecting 
projected  and  actual  cost  data  on  acquisition  programs  from  contractors 
and  in-house  government  plants  through  a single  integrated  system  for 
DOD  cost  analysis  and  procurement  management  purposes.  The  CMC  was 
to  provide  guidance  and  monitor  the  implementation  of  these  reports 
to  insure  consistent  and  appropriate  application  throughout  the  1X11)." 

( ;D 

Actual  test  of  the  CCDR  was  to  be  implemented  on  the  Trident 
Prog  nun.  It  might  be  interesting  to  note  that  the  PM  requested  this  be 
waived  but  it  was  denied  by  the  CMC. 


r 

\ 


*~v  ' 


THE  LEARNING  CURVE 


Learning  curves  are  a special  application  of  cost  estimating 
relationships.  They  are  a graphical/mathmatical  way  of  expressing  the 
process  of  learning  or  improving  as  a worker  completes  a repetitive  task. 
This  concept  of  learning  curves  is  based  upon  the  emperically  observed 
relationship  that,  given  a constant  production  environment  the  cumulative 
production  output  varies  inversely  with  the  manhour  input  labor.  The 
basic  use  for  this  type  of  estimating  is  to  forecast  the  unit  price  of 
an  item  or  system  over  its  production  life. 

Both  Industry  and  Government  have  found  that  the  unit  cost  of 
producing  hardware  decreases  as  the  cumulative  production  quantity  in- 
creases. To  quantify  this  relationship,  the  learning  curve  is  used. 

The  application  of  such  a curve  as  the  learning  or  progress  curve  is 
relevant  to  only  the  recurring  portion  of  production  costs  and  measures 
true  learning  only  when  all  recurring  costs  are  converted  to  constant 
dollars . 


While  there  are  several  different  hypotheses  about  the 
exact  manner  in  which  this  learning  01  cost  reduction  occurs, 
the  main  content  of  learning  curve  theory  is  that  each  time 
the  total  quantity  of  i touts  is  produced  doubles,  the  cost 
per  item  is  reduced  to  some  constant  percentage  of  its  pre- 
vious value.  If,  for  example,  the  cost  of  producing  the 
200th  unit  of  an  it  i is  SO  percent  • ’ the  cost  of  u \lucir,; 
the  100th  item,  the  cost  of  the  lOOth  unit  is  SO  percent  ol 
of  the  cost  of  200th,  and  so  forth,  then  the  production  pro- 
cess is  said  to  follow  an  SO  percent  unit  learning  curve.  Tf 
the  average  cost  of  producing  all  200  units  is  80  percent  of 
the  average  of  producing  the  first  100  units,  etc.,  then  the 
process  follows  an  SO  percent  cm  in lati ve  average  learning 
curve.  (3:8 1) 

The  two  most  familiar  form-,  of  learning  curves  assume  that  as  the 
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cumulative  production  quantity  of  units  doubles  the  cost  per  unit  de- 
clines by  some  constant  percentage.  One  form  treats  the  cost  as  the 
incremental  or  unit  costs;  the  other  treats  it  as  the  average  cost  for 
the  cumulative  number  of  units.  Either  formulation  results  in  a func- 
tion that  is  linear  when  plotted  on  logarithmic  grids.  When  the  in- 
cremental cost  assumption  is  made,  the  form  of  the  learning  curve  is 
termed  the  "log-linear  unit  curve",  when  the  average  cost  assumption  is 
made,  it  is  known  as  the  "log- linear  cumulative  average  curve." 

There  are  not  inherent  advantages  in  one  form  of  the  learning  curve 
as  opposed  to  the  other.  Where  it  is  possible  to  use  either  form,  the 
one  that  best  fits  the  data  should  be  used.  Sometimes,  however,  this 
option  may  not  be  open  because  of  the  nature  of  the  available  data. 

One  authority  has  suggested  that  experienced  manufacturers  and/or 
manufacturers  starting  production  with  major  engineering  problems  well 
under  control  are  likely  to  produce  along  a log-linear  unit  learning 
curve . It  is  further  suggested  that  new  manufacturers  and/or  those 
beginning  production  with  a large  number  of  engineering  problems 
unsolved  are  likely  to  produce  along  a log-linear  cumulative  average 
learning  curve. 

The  Log- Linear  Unit  Curve.  The  algebraic  expression  for  the 
log-linear  unit  curve  is: 

Yj ' =a\j 

where  Yj  = <-'ost  of  i unit 

a = cost  of  first  unit 
- cumulative  unit  number 
b = slope  parameter 
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The  parameters  a and  b may  be  estimated  directly  by  regression 
analysis  if  several  values  of  Y and  related  values  of  X are  known. 
However,  tins  is  usually  not  the  case.  Costs  of  hardware  items  are 
almost  always  accounted  for  in  "lots"  rather  than  in  single  units, 
and  the  average  unit  cost  of  a lot  is  not  the  cost  at  the  arithmetic 
mean  of  the  lot  units.  For  example,  if  a lot  includes  cumulative 
quantities  101  through  200  at  an  average  unit  cost  of  $100,  then  the 
cost  of  unit  150  is  not  $100.  This  is  because  learning  occurs  within 
the  lot  itself. 

Most  methods  of  obtaining  a match  of  cost  to  unit  number  involve 
estimating  the  unit  number  (the  algebraic  mid-point)  for  which  the 
average  lot  cost  is  applicable.  One  method  involves  the  use  of  tables 
from  which  the  algebraic  mid-point  may  be  calculated  for  a known  range 
of  unit  numbers  in  the  lot  and  an  estimate  of  the  learning  curve  slope. 
This,  of  course,  requires  an  iterative  process  if  subsequent  regression 
analysis  shows  that  the  slope  estimate  was  incorrect. 


DERIVATION  OP  CERs 


In  the  derivation  of  cost  estimating  relationships,  certain  steps 
should  be  followed:  The  first  is  the  consideration  of  the  various 

factors  that  may  cause  a given  result.  The  actual  determination  of 
caases  and  effects  of  past  experiences  has  to  be  ascertained.  The  next 
step  is  to  gather  data,  locate  interdependencies,  derive  a formal  state- 
ment of  the  relationship  and  establish  the  limitations  of  this  particular 
relationship.  This  is  considered  one  of  the  most  critical  steps  in 
parametric  cost  estimating,  for  it  is  in  collecting  the  data  that  an 
accurate  mathniatical  expression  can  be  developed  or  lost.  By  careful 
examination  of  this  data,  unexpected  cost  estimates  may  even  develop. 

Cost  analysts  arc  using  regression  theory  more  and  more  in  devel- 
oping cost  estimating  relationships.  In  regression  analysis  the  value 
of  one  variable  is  based  on  its  relationships  with  another.  The  theory 
of  regression  examines  whether  a relationship  exists  and  when  it  does 
the  nature  and  extent  of  the  relationship  is  measured. 

Therefore , the  essence  of  the  regression  analysis  is  that  func- 
tional relationships  between  cost  and  explanatory  variables  is  direct 
and  obvious.  A pattern  has  to  be  established  in  analysing  what  variables 
can  be  logically  related  to  cost.  Many  syst  today  seek  to  find  a 
direct  relationship  with  weight  and  cost.  However,  a close  analysis  of 
various  systems  will  show  this  is  not  always  possible. 

The  overall  estimate  must  have  the  following  characteristics: 
a.  an  unbiased  estimate  - the  weights  arrived  by  the  process  of 
regression  should  tiuely  repr  tent  the  rclat;  >n  of  that  vari  lo  to  the 
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tota]  cost.  Since  the  collection  of  all  sample  data  is  impossible, 
there  will  naturally  be  some  bias  in  the  weights. 

b.  an  efficient  estimate  - the  smallest  variance  in  a sample  would 
be  the  result  of  examining  all  the  possible  values  of  that  variable,  but 
to  tiie  extent  that  this  is  impossible,  the  estimate  will  be  inefficient. 

c.  a consistent  estimate  - again  this  is  a problem  of  sample  size 
in  that  the  larger  the  original  data  base,  the  more  consistent  will  be 
the  estimates. 

d.  a sufficient  estimate  - no  other  group  of  variables  weighed  in 
some  other  way  would  give  a better  estimate. 


USE  OF  PARAMETRIC  ESTIMATING  (CER) 


At  tliis  point  the  concept  of  parametric  estimating  as  it  is  used 
in  the  Army  (ECOM)  will  be  introduced.  Ibis  section  describes  a CER 
for  digital  computers  developed  by  the  Electronic  Command  (ECOM) 
located  at  Port  Monmouth,  New  Jersey.  ECOM  is  responsible  for  managing 
the  entire  life  cycle  development,  procurement,  and  fielding  of  tactical 
communication,  avionics,  radar,  automatic  data  processing,  meterology, 
night  vision,  combat  surveil lance,  target  acquisition,  navigation 
and  electronic  warfare  equipment  and  systems. 

The  ECOM  digital  computer  CEU  was  composed  of  commercial  third 
generation  general  purpose  computers,  ranging  in  capacity  from  small  to 
large.  In  the  derivation  of  cost  data  ECOM  found  much  inconsistency 
although  their  statistical  methodology  was  correct.  Appropriate  cost 
adjustments  were  made  and  additional  computer  characteristics  were 
calculated  and  two  statistically  significant  CERs  were  derived  for 
predicting  the  acquisition  cost  of  a digital  computer:  one  based  on  a 
sample  of  11  computers  and  the  other  based  on  the  same  11  computers  with 
two  different  core  storage  sizes  for  each,  i.e.,  a sample  size  of  22. 

POPULATION  DEITMTION 

The  two  CERs  base'  on  the  acquisition  ce-;t  of  the  central  pro- 
cessing unit  with  floating-point  arithmetic,  console,  and  core  storage 
of  commercial  general  [impose  third-generation  computers  ranging  in  size 
from  small  to  large  scale. 


DATA  COLLECTION,  ANALYSIS,  AND  ADJUSTMF.NT 

The  first  step  which  DCOM  assumed  was  to  list  all  variables  that 
they  expect  would  be  closely  l'elated  to  cost.  These  variables  included 
everything  from  weight  to  actual  volume  size  of  components.  After 
gathering  this  large  volume  of  data  they  searched  for  variable  inter- 
dependencies with  cost.  The  following  variables  were  considered  per- 
tinent and  hence  they  were  collected  for  each  computer. 

1.  fixed  point  add  time 

2.  words  of  core  storage  (in  terns  of  bits  by  multiplying  number  of 

words  by  bits  per  word.  This  makes  core  storage  comparable 
among  computers  with  different  word  sizes. 

3.  bits  per  word  (calculated  by  dividing  cycle  time  into  total 

number  of  bits  accessed  per  memory  cycle.  This  places  com- 
puter that  access  different  numbers  of  bits  per  memory  cycle 
on  a more  comparable  basis) 

4.  nemory  cycle  time 

5.  number  of  operation  codes 
PRfh  I Ml. VARY  .ANALYSIS 

During  their  next  step  in  seeking  to  establish  a relationship  in 
order  to  develop  a mathmetical  expression,  a graphical  analysis  of  cost 
vs  core  storage  size  indicated  that  these  were  linearly  related,  though 
not  sufficient  to  produce  a satisfactory  Cl-R. 

By  examining  values  of  other  characteristics  and  relating  these  to 
the  cost  differences  not  accounted  for  by  the  core  sire  (c.g.,  projected 
costs  of  computers  with  one  mi  11  ion  hits  of  storage)  it  appeared  that 
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inclusion  of  two  additional  characteristics-bits  per  memory  cycle  and 
fixed-point  add  time-would  result  in  an  improved  relations,  (variables 
1,  3 f,  4 above) 

Algebraic  expressions  were  formulated  using  these  three  variables, 
and  regressions  were  run  on  an  ECOM  computer.  Next,  analyst  tried 
different  combinations  of  the  variables  (both  linear  and  log- linear)  to 
see  which  one  gave  the  best  predictions  based  on  past  data.  The  best 
results  were  obtained  with  the  linear  formulation  of  core  storage,  bits 
per  memory  cycle  time,  and  the  reciprocal  of  fixed-point  add  time.  It 
may  be  noted  that  linear  formulations  usually  present  the  best  results. 

CER  RESULTS 

The  CERs  developed  are  given  below.  Tie  first  CER  is  based  on  one 
core  storage  size  for  each  computer: 

C = 105  + 2.90  A + .157  S + 709  (I  ) I 

where  C =*  costs  in  thousands  of  dollars 

A = number  of  bits  accessed  per  memory  cycle  time  (bits 
per  microscc) 

S = core  storage  size  (thousands  of  bits) 

1 2~  fixed-point  add  time  (microseconds) 

and  R“=  . 96 

standard  error  of  estimate  ~ 84, 

F ratio  = 28.6 
Sample  size  = 1 ' 

The  second  CER  is  based  on  two  core  storage  sizes  for  each  o : mUv 


- 


C = 93  + 3.38  A + .153  + 698  (j) 
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where  the  variables  are  defined  as  earlier, 
and  R2  = .97 

Standard  error  of  estimate  = 90 

F ratio  = 108.8 

Sample  size  = 22 

2 

The  reasons  for  selecting  this  as  best  is  that  R has  the  highest 
value.  R2  being  defined  as  the  coefficient  of  determination  of  the 
variables. 

Although  hqs.  I and  II  are  statistically  valid,  they  apply  only  to 
computer  technology  as  of  about  1965.  Since  that  time  many  advances 
have  been  made  that  have  the  effect  of  increasing  performance  for  a given 
price  or  decreasing  price  for  a given  performance  value  or  core  storage 
si  ze. 

A possible  approach  to  developing  ChRs  that  could  be  used  for 
predicting  future  computer  costs  would  lie  to  analyze  cost  and  charac- 
teristic data  representing  computers  over  several  years  and  technologies* 
This  would  add  more  data  points  than  was  possible  during  the  current 
effort . 
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COST  hSTOlATIXG  COMPARISON'S 

'Hie  two  main  categories  of  estimating  are  judgemental  and  scientific 
estimating.  In  judgemental  estimating,  the  estimates  are  made  subjec- 
tively with  or  without  an  explicit  procedure  on  the  basis  of  intuition, 
experience,  and  common  sense.  The  distinguishing  feature  of  judgemental 
estimating  is  its  lack  of  uniformity  as  the  estimates  vary  with  the 
estimator.  This  is  not  to  say,  however , that  judgemental  estimations 
will  provide  results  inferior  to  those  produced  by  other  methods.  A 
common  sense  estimate  or  intuitive  guess  by  an  expert  may  be  the  best 
possible  (or  luckiest)  estimate.  Scientific  estimating  is  sometimes 
categorized  into  time  series  (projection)  and  parametric,  models. 

In  the  time  series  estimating  method,  the  emphasis  is  on  pattern 
and  pattern  changes,  and  the  method  thus  relies  entirely  on  historical 
data.  Parametric  estimating  on  the  other  hand,  seeks  to  develop 
relationships  among  system  elements  by  using  refined  and  specific 
relationships,  thereby,  being  powerful  enough  to  take  special  events 
favorably  into  account.  Ko  one  iias  ever  constructed  a parametric  model 
that  can  guarantee  success,  except  on  a probabilistic  basis. 

The  time  series  analysis  and  projection  type  of  estimate  cover  many 
different  methods  and  techniques.  The  basic  thrust  of  all  of  these 
method,  is  that  they  attempt  to  extrapolate  the  behavior  of  a time  series 
by  relying  wholly  on  past  data  and  not  searching  for  any  parametric  agents. 
Because  of  this,  time  series  methods  are  sometimes  called  naive.  One 
of  the  more  well  known  time  series  methods  of  estimating  is  Curve  Pitting. 

This  method  is  the  simplest  and  rest  cnazonly  used  r.  thod  of 
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time  series  analysis.  'Hie  objective  of  curve  fitting  is  to  select  an 
analytical  model  that  fits  the  time  series  fairly  well.  In  general,  the 
model  contains  parameters  that  arc  unknown  and  mast  be  selected  such 
as  to  give  a "best"  fit,  where  "best"  is  a specified  condition. 

There  is  no  doubt  that  parametric  (CF:R)  or  what  is  referred  to  as 
regression  analysis  can  be  put  to  great  use  in  cost  estimating.  However, 
CbRs  do  have  limitations  which,  along  with  their  advantages,  should  be 
recognized.  CHRs  cannot  capture  all  of  the  variances  in  cost,  therefore, 
they  can  provide  only  a ballpark  figure. 

Advantages  of  CHRs  include: 

a.  Less  expensive  to  obtain  than  "grass  roots"  estimates  for  any 
relatively  large  or  complex  system. 

b.  Statistics  related  to  the  Cl  ill  give  some  indication  of  the 
credence  to  be  placed  on  the  estimate.  Summary  figures  of  merit  are  more 
difficult  to  generate  for  the  industrial  engineering  estimate. 

c.  A C! .11  is  a relatively  objective  method  of  estimating  cost^ 
compared  to  estimating  by  analogy. 

d.  The  QiR  approach  is  a no re  sensitive  method  of  estimating  than 
the  analogy  approach  in  that  the  former  is  based  on  an  "average"  of 
several  observations. 

c.  Statistics  related  to  a CF.R  give  a viewer  some  indication  of 

the  merit  of  the  CI-R. 

Disadvantages  of  Q-Rs  include: 

a.  Since  industrial  engineerin’  estimates  are  based  on  specific 
production  process  and  materials,  il  • -evs  reasonable  to  expect  the 
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industrial  engineering  approach  to  produce  more  accurate  results  than  the 
CF.R.  Although  it  is  possible  to  use  both  types  of  estimates,  parametric 
is  about  all  that  is  available  during  the  Conceptual  and  Validation  phase, 
since  there  isn't  much  data  available.  However,  once  the  Full-Scale 
Development  phase  commences,  grass  roots  estimates  can  be  made  using 
the  data  within  the  Work  Breakdown  Structure  (WBS)  packages. 

b.  When  a system  has  few  or  no  historical  counterparts  it  may  not 
be  a practical  alternative  to  use  CFiRs. 

c.  In  comparison  to  the  analogy  approach  which  involves  the  relation 
of  cost  to  a single  variable,  the  CFR  approach  requires  collection  of 
data  from  several  predecessors. 

d.  Although  data  on  earlier  items  or  systems  may  have  once  been 
readily  available,  the  data  may  have  later  been  destroyed  in  such  a 
manner  as  to  be  virtually  unobtainable. 

e.  Data  might  be  available  but  only  at  very  high  costs. 
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UNCERTAINTY 


"Uncertainty  relates  to  those  phenomena  which  create  variability 
and  cannot  be  insured  against  by  adding  to  costs."  (6:1)  It  is  the 
result  of  a subject  reasoning  process  and  refers  to  the  unspecification 
of  a certain  degree  of  ignorance. 

The  difference  between  original  estimates  and  final  costs  of  weapon 
systems  have  been  so  great  that  as  of  late,  agencies  have  been  looking 
into  various  case  histories  in  an  attempt  to  identify  reasons  for  cost 
overruns.  These  efforts  were  certainly  justified  as  the  increase  in 
cost  for  various  systems  are  considerable.  For  example,  the  factor 
increase  in  fighter  aircraft  ranged  from  "on  cost"  to  3.9.  Bombers 
ranged  to  6.2,  cargo  aircraft  were  less  prone  to  overrun  with  a factor 
of  1.5  and  missiles  depicted  a wide  variation  which  ranged  to  14.7 

The  main  point  of  interest  in  these  overruns  is  the  reasons  for 
t lie  increases  and  whether  the  increases  are  due  to  bad  or  incorrect  cost 
estimating. 

Studies  have  concluded  that  the  main  reason  for  the  above  increases 
was  due  to  cost  estimating  uncertainty.  This  refers  to  variations  in 
cost  estimates  of  a system  for  which  the  configuration  is  essentially 
fixed.  Specifically,  the  factors  associated  wi th  uncertainly  arc: 

a.  lirrors  in  cost-estimating  relationships. 

b.  l-.rrors  in  data. 

c.  1 extrapolation  errors. 

Some  examples  of  uncertainty  include: 

a.  Origin,  time,  and  extent  of  fa:  :u  threat  to  cur  nation.  . 


security. 


b.  Capability  of  a particular  weapon  system  to  meet  a threat  with 
regard  to  equipment  performance,  reliability,  and  operation. 

c.  Capability  of  cost  analysis  to  translate  the  design  of  the 
weapon  system  into  a statement  of  resource  requirements  and  costs. 

Proposals  for  treatment  of  uncertainty  in  cost  analysis  range  from 
conventional  statistical  tools  to  the  application  of  what  are  commonly 
known  as  "fudge  factors."  Hie  latter  is  ruled  out  because  of  a high  sense 
of  morality  that  states  that  fudge  factors  arc  wrong. 

Conventional  statistical  tools,  are  of  only  limited  value  in  coping 
with  uncertainty  in  cost  analysis  because  of  the  occasions  for  which  they 
can  be  used  rigoriously  are  quite  rate.  Even  in  these  cases,  we  may 
still  have  problems  because  of  difficulty  in  justifying  the  assumptions 
of  the  statistical  model  in  our  particular  application. 

finally,  where  conventional  statistical  methods  are  applicable,  it 
is  most  likely  to  be  in  the  treatment  of  cost- estimating  uncertainty 
rather  than  requirements  uncertainty  which  is  the  more  important  of  the 
two . 

One  of  the  most  useful  tools  for  dealing  with  uncertainty  is  the 
technique  of  Cost  Sensitivity  Analysis.  The  primary  objective  of  this 
type  of  analysis  is  to  provide  a system  designer  or  a planner  with 
insights  into  the  way  system  costs  arc  influenced  by  changes  in  assump- 
tions and  specifications.  The  Cost  Sensitivity  \nalysis  appears  promising 
because  it  highlights  the  uncertainty  inherent  in  future  system  costs 
and  gives  the  planner  a full  view  of  the  o'-t  implication  on  decisions 


affecting  system  configuration  and  operations.  In  general,  tnis  type 
analysis  is  the  process  of  examining  how  total  system  resource  require- 
ments or  costs  change  as  key  system  characteristics  are  varied  over  a 
relevant  range.  A cost  analyst  can  point  out  where  costs  are  relatively 
sensitive,  discover  ranges  of  values  within  which  costs  do  not  change 
significantly  and  outside  of  which  costs  rise  sharply,  and  provide  some 
meaningful  answers  to  problems  by  using  the  Cost  Sensitivity  Analysis 


Another  technique  of  dealing  with  uncertainty  is  the  use  of  the 
judgemental  forecast  called  the  Delphi  Method.  Here  a panel  of  experts 
is  interrogated  with  a sequency  of  questions  (or  questionnaires)  to 
which  the  next  question  (or  questionnaires)  is  based  upon  the  previous 
response.  Thus,  all  the  experts  are  given  access  to  all  the  information 
for  forecasting.  This  method  proports  to  minimize  the  "bandwagon  effect" 
of  public  opinion. 

Another  method  for  dealing  with  uncertainty  begins  with  the  assump- 
tion that  a cost  analyst  can  describe  each  input  parameter  with  a prob- 
ability distribution.  This  distribution  is  then  treated  as  a theoretical 
population  from  which  random  samples  are  obtained.  The  methods  of  taking 
such  samples,  as  well  as  problems  associated  with  these  techniques  arc 
referred  to  as  tin  "Monti  Carlo"  Method. 


The  Monte  Carlo  technique  involves  constructing  a 
cumulative  probability  distribution  free  the  probabil- 
ity distribution  dcsei ibing  the  actual  or  estimated 
input  uncertainty.  Next,  a ra  idem  decimal  between 
zero  and  one  is  selei  : ?d  :T-  : ' ihle  rand  di  .its. 

As  presented  in  (bait  7,  by  projectin'-  horizontally 
from  the  point  on  the  Y-nxi.-  cone  ■>  . line,  to  the 
random  decimal  to  Lh  - a:  il..t  ivv  run  , \e  fit  ! the 


value  of  x corresponding  to  the  point  of  intersection. 

This  value  is  taken  as  a sample  of  the  value  of  x,  the 
input  parameter.  The  result,  if  this  procedure  is  re- 
peated numberous  times,  is  a sample  of  input  value  that 
approximates  the  required  uncertainty.  (6:5) 

It  must  be  remembered  while  preparing  any  cost  estimations  there 
is  always  an  end  to  available  data.  However,  in  many  instances  there 
is  a need  to  go  beyond  the  data  base  and  extrapolate  further,  iihen  this 
is  done  our  confidence  decreases  (uncertainty  increases  as  illustrated 
in  Chart  8).  This  illustrates  how  additional  cost-estimating  arises  when 
we  are  forced  beyond  the  data  base. 

In  sumnary,  cost  analysis  of  future  military  capabilities  are 
subject  to  both  stat.e-of- the-world  uncertainties  as  well  as  statistical 


uncertainties. 
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Although  cost  overruns  have  occurred  for  centuries  (e.g.,  a Roman 
aqueduct  cost  more  than  twice  the  original  estimate),  they  are  exclusive 
to  neither  government  nor  military  and  are  not  limited  to  projects 
advancing  the  state  of  the  art.  There  is  a definite  need  for  these 
excesses  within  the  acquisition  of  major  systems  in  the  military  services 
to  he  reduced  in  scope.  This  is  to  be  accomplished  by  primarily  improving 
the  process  of  acquiring  major  weapon  systems.  The  acquisition  process 
has  been  the  subject  of  analysis  and  police  recommendations  for  years, 
yet  evidence  of  improvement  is  only  noticeable  during  the  past  few  years. 

It  seems  that  cost  growth  is  directly  related  to  and  the  result  of 
adjustments  in  target  goals.  Although  adjustments  are  a necessity  in 
many  instances,  they  should  be  held  to  a minimum.  Research  has  indicated 
the  need  for  a continuing  effort  to  control  the  cost,  schedule,  and 
performance  outcome  of  programs  and  for  better  understanding  of  the  causes 
of  program  cost  growth. 

System  acquisition  policy  should  be  flexible  but  based  on  in- 
cremental acquisition  strategies  as  the  normal  approach  for  the  1970s. 

In  particular,  development  should  be  separated  from  subsequent  pro- 
duction, and  the  initial  ; union  of  development  should  concent  rat  on 
demonstrating  system  performance  ami  be  conducted  in  a healthy  austere 
fashion. 

It  is  believed  that  the  methods  used  in  parametric  cost  estimating 
an  a sensible  way  of  usin’  an  imper 1 ect  information  about  the  past  to 
decrease  the  uncertaint ii  about  future  weapons  cost.  Par. metric  cost 
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analysis  or  what  is  commonly  referred  to  as  regression  analysis  involves 
finding  physical  or  program  characteristics  that  have  a relationship  to 
cost.  Such  relationships  could  include  weight,  speed,  power  and  its 
impact  upon  cost.  j] 

further,  one  of  the  most  effective  ways  to  strengthen  cost  controls 
is  to  reduce  the  task  uncertainty  associated  with  cost  estimating. 

In  conclusion  it  must  be  remembered  that  estimating  cost  relation- 
ships arc  very  complex  and  while  statistical  analysis  can  aid  in  providing 
a understanding  of  factors  which  influence  cost,  parametric  analysis  does 
not  solve  all  problems  in  cost  growth.  The  use  of  cost  estimating 
relationships  should  be  utilized  as  an  aid,  to  be  supplemented  by  expe- 
rience, judgement  and  other  types  of  information. 
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36 


1 1 


i1 


1 


Deportment  Cf  Defense  Directive.  Subject:  OSD 

Cost  Analysis  Improvement  Group  ( 5000.4) . June 
13,  1S73. 


This  directive  provides  a permanent  charter  for  the 
CSD  Cost  Analysis  Improvement  Group  (CA1G). 

Department  of  Defense  Instruction.  Subject:  Contractor 

Cost  Data  Reporting  (CCDR).  September  5,  1973. 

This  instruction  provides  guidance  for  collecting 
projected  and  actual  cost  data  on  acquisition  program 
from  contractors  and  in-house  government  plants  through 
a single  integrated  system. 

’memorandum  for  the  Secretaries  cf  the  military  De- 
partments. Subject:  Cost  Estimating  for  Vapor 

Defense  Systems . January  33,  1972. 

This  memorandum  states  that  the  secretaries  of  the 
military  services  should  utilize  the  parametric 
aHproach  in  their  cost  estimating  procedures. 


Operations  Research/Systems  Analysis  Executive  Courss 
Cost  Estimating  Relationships,  Fort  Belvoir. 


A lesson  outline  introducing  the 
Estimating  Relationships  used  at 
search/Sys terns  Analysis  Executive 


subject  of  Cost 
the  Operations  Re- 
Course. 
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